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ABSTRACT 



The English Language Proficiency Test (ELPT) is a 



multiple-choice examination that is designed to assess the test taker's 
ability to use English in day-to-day interactions involving listening and 
reading. It is intended primarily as an admissions and placement test for 
college students with English as a second language. The ELPT consists of 
subtests for listening skills and reading skills. Research generally 
supporting the validity of the ELPT was reviewed, and the external aspects of 
construct validity were studied with a special data collection and analyses. 
One set of analyses addressed the relationship of proficiency ratings as made 
by the ELPT to proficiency ratings made by students ' teachers using the same 
scale descriptors. The second set of analyses investigated the relationship 
of ELPT scores to college grades assigned in English as a second language 
courses, regular English classes, and/or freshman grade point average (GPA) . 
Two samples were used, one of 412 high school students from 32 classes and 24 
schools and the other of 190 college students from 15 classes over 10 
colleges. In the college sample, ELPT reading standard scores correlated 0.50 
with teacher ratings of reading proficiency and 0.48 with teachers' relative 
rankings of reading competence. In the high school sample, comparable 
correlations were 0.68 and 0.69. In the college sample, the correlation for 
listening scores was 0.57 with teacher ratings of proficiency and 0.56 with 
teachers' rankings. In the high school sample, these ratings were 0.71 and 
0.67 respectively. For the 2 colleges for which GPA was available, the 
reading correlation was 0.53 for 1 college, and 0.05 for the other (perhaps a 
function of small sample size and relatively high reading scores) . Results 
for the listening scale also suggest that the kinds of language skills 
assessed by the ELPT play some role in overall academic success, but are 
hardly deterministic of success or failure. Yet to be investigated is whether 
the absence of writing or speaking components of the ELPT is important in 
assessing the usefulness of the measure. An appendix defines the reading and 
listening proficiency scales. (Contains eight tables and two references.) 
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The English Language Proficiency Test (ELPT) is a multiple-choice examination 
that is designed to assess the test taker’s ability to use English in day-to-day interactions 
involving listening and reading. Thus, it emphasizes functional, practical language. It is 
intended primarily for use as an admissions and placement test in two- and four-year 
colleges for students with English as a second language. The primary target population is 
high school students who have lived in the United States for at least two years and who 
have either come from a country whose primary language is not English or who come 
from homes where English is not the principal language. It may also be useful for students 

m English as a second language classes regardless of how long they have been in the 
United States. 

The ELPT consists of two subtests: one attempts to measure listening skills, and 
the other, reading skills. Each of these two subtests consists of about 42 items which are 
to be completed in thirty minutes, for a total test time of one hour. 

Separately for the listening and reading subtests, the ELPT categorizes students 
into one of five proficiency levels. These levels are intended to provide descriptions of 
what students categorized in each level can do. The levels (below intermediate, 
intermediate, intermediate high, advanced, and advanced high) are defined in the 
Appendix. In addition, scaled scores are provided. The Reading and Listening scales run 
from 1 to 50 and the Total scale runs from 901 to 999. (These scales were selected to 
avoid confusion with the 200-800 scale used to report scores for the SAT I: Reasoning 
Tests and SAT II: Subject Tests.) In the pamphlet. “Understanding Scores from the 
English Language Proficiency Test” scaled scores are defined as follows: 
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This paper presents some evidence related to the validity of the ELPT. As 
Messick (1996, p. 6) has observed, “Validity is » overall evaluative judgment of the 
degree to which empirical evidence and theoretical rationales support the adequacy and 
appropriateness of interpretations and actions based on test scores or other modes of 
assessment” (bold in the original). Messick focuses on construct validity, noting “score 
meaning is a co nstruction that makes theoretical sense out of both the performance 
regularities summarized by the score and its pattern of relationships with other variables” 

(p. 6). Although validity is seen as a unified concept, i, is useful to separately consider the 
six aspects of construct validity identified by Messick; these are content, substantive, 
structural, generalizability, external, and consequential. 

Content 

From its inception, the ELPT was designed as a proficiency test; a proficiency 
scale was NOT simply appended to an existing measure. Thus, such topics as grammar 
and usage .ha, are frequently found on achievement tests receive much less emphasis on 
the ELPT which concentrates on assessing practical use of the English language. Test 
development was driven by con.emporaty theories of function* language use in both 
academic and non-academic settings as interpreted by a committee of expens both internal 
and external to ETS The external members of the committee included ESL teachers. 



e 0 e administrators, and college faculty with expertise in the assessment of English for 
speakers of other languages. These and other external experts who reviewed an early 
version of the test suggested that more emphasis should be placed on the use of English in 
academic settings, and these changes were incorporated in the final version of the ELPT. 

Substantive 

This aspect of construct validity emphasizes the need for assessment tasks to 
sample domain processes (not just domain content) and to provide evidence that 
“ostensibly sampled processes are actually engaged by respondents" (Messick, 1996, p. 

10). Inspection of the test form suggests that reading and listening processes are indeed 
engaged by the ELPT. It appears that reading processes must be engaged to answer the 
reading questions, and listening processes (plus reading processes) must to engaged to 
answer the listening questions. For one question type in the Listening section (rejoinders), 
both the question and answer choices are presented on audio tape, and the examinees need 
only mark the appropriate letter on the answer sheet. For the other question type in the 
listening section (dialogues), the examinees hear a selection such as a dialogue, an 
announcement, a news report, or a narrative and then read and answer a multiple-choice 
question based on the selection that they just heard. As might be expected, scores from 
the dialogue question type showed a higher correlation with the total reading score (r = 

.81) than did scores from the rejoinder question type (r = .69). Corrected for unreliability 
of both the listening and reading scores, these correlations were .92 and .84 respectively. 
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Additional studies are needed to determine the extent to which apparently 
necessary skills are actually tapped by the test questions. In particular, luture studies 
should determine whether students can answer the questions without reading the passages 
or listening to the recordings of the listening tasks. 

Structural 

Messick (1996) notes that “the internal structure of the assessment (i.e., 
interrelations among the scored aspects of task and subtask performance) should be 
consistent with what is known about the internal structure of the construct domain" (p. 

1 1). Based on data from the first operational administration of the ELPT in November of 
1995. the coefficient alpha reliability of the Reading score was .91 and the reliability of the 
Listening score was 89. The correlation of the Reading and Listening scores was .83. 
Corrected for unreliability, the correlation of the two scores was .91. As expected, 
reading and listening skills are highly related but the corrected correlation is still less than 
1.0. The corrected correlation was somewhat higher than the correlation between similar 
scores in the Test of English as a Foreign Language (TOEFL) in which corrected 
correlations in the low to mid ,80's across five major language groups have been reported 
(Hale, Rock, & Jirele, 1989). Because students in the ELPT sample resided in the United 
States for at least two years, they were immersed in both reading and listening tasks and 
would tend to learn these skills together, the foreign students in the TOEFL sample may 
experience greater variability in the extent to which reading or listening tasks are 
emphasized in their academic English programs. 




6 



4 



Further evidence of discriminant validity is provided by the correlation of the 
ELPT scores with the verbal score from the SAT I: Reasoning Test (SAT I-V). A major 
component of the SAT I-V is a reading test; about half of the questions relate to reading 
passages. There is no listening section. Thus, the ELPT Reading score should be more 
highly related to SAT I-V than the Listening score. This was the case with correlations 
with SAT I-V of .75 and .69 for Reading and Listening respectively. Also as should be 
expected, correlations with the SAT I mathematics score (SAT I-M) were substantially 
lower (.49 and 43 for Reading and Listening respectively). The ELPT Total score was 
correlated .76 with SAT I-V and .51 with’ SAT I-M. 

Generalizability and External 

Evidence of ...generalizability depends on the degree of correlation of the 
assessed tasks with other tasks representing the construct or aspects of the construct” 
(Messick, 1996, p. 1 1). Evidence could come from correlations with other multiple- 
choice assessments in the same domain, but much stronger evidence comes from noting 
relationships to criterion performances that are measured in quite different ways. With 
such catena, the external aspect of construct validity can be subsumed under the same set 
of analyses. We addressed these issues with a special data collection and analyses that are 
described below. The first set of analyses addressed the relationship of proficiency ratings 
as made by the ELPT to proficiency ratings (using the same scale descriptors) made by the 
students’ teachers. The second set of analyses investigated the relationship of ELPT 
scores to college grades assigned in English as a second language classes, regular English 
classes, and/or freshman grade point average (GPA). Many factors besides language 



ab,l,ty influence course grades, thus the correlations for these analyses would be expected 
to be substantially lower than for the first set analyses in which both predictor and 

e clearly attempting to assess the same type of language skills and abilities. 
Analyses of Teacher Ratings 

Sample. Two samples were used, a sample of high school students and a sample 
dents enrolled in two- or four-year colleges. For both samples, students in English 
as a second language (ESL) classes were targeted. Recruitment letters were sent to ESL 
teachers from a regionally and economically diverse set of institutions. Participating 
teachers agreed to administer the ELPT and to independently rate the listening and reading 
proficiency of their students. Students who were no. in the prima* targe, population for 
the ELPT were screened from the sample. Because analyses were conducted within 
classrooms and then averaged across classrooms, classes with fewer than four students 
meeting the eligibility cnteria (in target population, complete ELPT scores, and complete 
teacher ratings) were eliminated. This resulted in a final sample of 190 college students 

(from 15 classes spread over 10 colleges) and 412 high school students (from 32 classes 
spread over 24 high schools). 

Materials. Sample rating sheets, with scale definitions, are provided in Appendix 

B. 

Procedures. Teachers were asked to first complete the Listening Rating Form and 
then the Reading Rating Form. The teachers were instructed as follows: 

On each form you will do two ratings for each student. The proficiency 
rating will evaluate your students with respect to a defined standard. The relative 



ranking will evaluate your students with respect to each other. Because the 
proficiency ratings are on a predefined scale, you may find that you are using some 
score points more than others or that some score points are not used at ail; this is 
perfectly appropriate. On the other hand, the number of students in each category 
in the relative rankings should be balanced. 

For the proficiency ratings, read over the attached definitions, then circle 
the letter (or + for intermediate high) that corresponds to your evaluation of the 
student’s proficiency. 

For the relative rankings, each student you are rating should be compared 
to the other students you are rating and assigned a rating as top quarter (1/4), 
second quarter (2/4), third quarter (3/4) or bottom quarter (4/4). Rankings should 
be relative to the other students your are rating on the Rating Form, not relative 
to all of the other students in your classes (unless ail of your students appear on 

the Rating Form). The number of students in each quartile should be as equal as 
possible.... 

Because scores are considerably more variable across courses than within courses, 
and because the ELPT is intended for use in unselected groups (in order to make 
admissions or placement decisions), within course correlations were corrected for 
restriction in range. Gulliksen’s (1950, p. 137) equation 18 was used, with the standard 
deviation of scores in the unselected population estimated from the total across course 
standard deviations for the reading and listening scores (10. 1 for reading and 10.8 for 
listening). These withm-course corrected correlations were converted to zs, weighted by 
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aVeraged ’ and convened back i"‘° a correlation coefficients In addition to these 
averaged, corrected correlations, crosstabulations of test-assigned and teacher-assigned 
proficiency ratings were computed. 

Res"t,s. In the college sample, ELPT reading standard scores correlated .50 with 
teacher ratings of reading proficiency and .43 with teachers' relative rankings of readin. 
competence. In the high school sample, the comparable correlations were .68 and. 69. In 
the college sample, the correlation for ELPT listening standard scores was .57 with 
teacher ratings of proficiency and .56 with teachers* rankings. In the high school sample, 
the correlations were .71 and .67 for proficiency ratings and relative rankings respectively. 

The crosstabulations of teacher ratings and ELPT proficiency scores for both 
reading and listening, separately for the high'school and college samples, are presented in 
Tables 1 to 4. (Ratings of listening proficiency were made first; some teachers did not 
complete the reading ratings, so sample sizes were slightly higher for the listening ratings.) 
Because results were quite consistent across both samples and both types of proficiency, 
we will discuss only Table 4. The clustering of scores along the diagonal confirms the 
relatively high correlation observed between test scores and teacher ratings, but it is also 
apparent that teachers generally report higher proficiency levels than the test scores 
suggest. For example, the test assigns more than three times as many students to the 
Below Intermediate (L) level than the teachers, and the test assigns almost twice as many 
to the Intermediate (I) level. Teachers assigned five times as many students to the 
Advanced High (H) level as the test. There are 140 students along the five cells on the 
main diagonal of the table, indicating exact agreement between proficiency ratings 
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assigned by the test and the teachers. However, just below the main diagonal (indicating 

teacher ratings that are one category higher than test ratings) there are 204 students in just 
four cells. 

Although these results suggested that cut scores for each proficiency level on the 
test may be too high, lowering the cut scores was rejected for two reasons. First, trainers 
who were experienced with teaching language teachers to make proficiency ratings noted 
that at the initial stages of training naive raters tend to rate about one category too high 
(Rabiteau, personal communication). The teacher raters in this study were not exposed to 
any formal training, and had to rely only on the written descriptions of the proficiency 
categories. Second, a Nedelsky cut score study suggested that the existing cuts were not 
too high and may even be too low. In the Nedelsky study, five experts (three ETS staff 
members and two outside linguists) rated each distractor to determine whether a minimally 
competent student at each proficiency level could eliminate the distractor. The three ETS 
staff worked together and provided a single consensus ratihg while the two external 
consultants both worked independently. Thus, this procedure generated three independent 
estimates of a cut point for each level on the reading and listening scales. For only one cut 
point (Advanced High on the reading scale) was any of these three estimates lower than 
projected from the teacher ratings and most were substantially higher. 

Analyses of Course Grades and GPA 

Sample. One of the community colleges in the teacher rating sample also 
provided GPA data. In addition, one four-year college that did not provide teacher ratings 
supplied grades in regular English courses and GPA. One community college that 
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provided data in the teacher ratings sample conducted a second round of testing on a 

different set of students; ESL course grades, but no teacher ratings, were provided for this 
sample. 

Procedures. Teachers provided mid-term grades in the ESL courses or regular 
English courses. These grades were on an F-A scale with some also containing plusses 
and minuses. Grades were convened to a 0-4 numerical scale as follows: F = 0,0, D- = .7, 
D = 1.0, D+- 1.3, C- = 1.7, C = 2.0, etc. The ELPT was administered in these classes 
within a few weeks of when the mid-term grades were assigned; teachers did not have 
access to the ELPT scores before assigning grades. Grades were correlated with ELPT 
standard score for reading, listening, and total. Correlations were corrected for range 
restriction on the Reading or Listening scores but not on the criterion scores. 

Results The means, standard deviations, and corrected correlations with English 

course grades (ESL for College 1 and regular freshman composition for College 2) for the 

ELPT Reading score are presented in Table 5. The lower correlation observed in College 

2 may be a function of both the small sample size and the relatively high Reading scores 

that may not discriminate well at the upper end. Also note that the data for College 1 is 

based on the concurrent correlation with grades in an ESL class while College 2 data is 

based on correlations with grades in an English composition class that is open to all 

students. Data from additional colleges is needed before these relationships can be well 
understood. 
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Table 6 provides comparable data for the ELPT Listening score. Results 
essentially mirrored the results for the Reading score with a relatively substantial 
correlation in College I and a lower correlation in College 2. 

Tables 7 and 8 show the relationships with college GPA for the Reading and 
Listening scores respectively. These results suggest that the kinds of language skills 
assessed by the ELPT play some role in overall academic success but they are hardly 
deterministic of either success or failure. 

Consequential 

Messick (1996) suggests that the consequential aspect of construct validity 
“includes evidence and rationales for evaluating the intended and unintended consequences 
or score interpretation and use in both the short- and long-term, especially those 
associated with bias in scoring and interpretation, with unfairness in test use, and with 
positive or negative washback effects on teaching and learning” (p. 12). It is too early to 
assess the positive or negative washback effects of the ELPT on teaching and learning. 
Teachers should be surveyed to determine whether they have modified any teaching 
practices to prepare students for the ELPT. If they have, these practices should be 

reviewed by experts to identify those which are on balance positive and those which are 
essentially negative. 

Messick warns that construct underrepresentation can threaten the validity of an 
assessment; this may occur if some important aspect of criterion performance is not 
included m the assessment. Thus, for example, a language proficiency test that focused 
solely on reading skills would underrepresent the listening, writing, and speaking skills that 
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also may be very important aspects of communicative competence The ELPT does 
include an assessment of listening skills, but not speaking or writing skills. Information on 
these skills would need to be obtained from other sources if they are deemed to be a 
necessary part of a comprehensive assessment. However, in many academic settings, 
especially in large sections of freshman-level courses, speaking and writing skills may be of 
only minimal importance. In other courses, these skills may be more critical. Because a 
test per se is not validated, but rather the use of the test for a particular purpose, the 
importance of the absence of a speaking or writing component in the ELPT can only be 

judged in the context of the how the score will be used and what additional evidence might 
be submitted along with the ELPT scores. 
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table i 




Reading Proficiency 



ELPT Reading 
Proficiency Score 




H 


A 


+ 


I 


L 


Row 

Total 


Advanced High 


(H) 


1 


2 


0 


0 


0 


3 


Advanced 


(A) 


4 


6 


3 


0 


0 


13 


Intermediate High 


(+) 


14 


34 


19 


9 


2 


78 


Intermediate 


(I) 


8 


20 


30 


19 


5 


82 


Below Intermediate 


(L) 


0 


6 


8 


11 


4 


29 



TABLE 2 



EL p T Reading Proficiency Scores by Teachers Ratings of 
^Reading Proficiency for High School Sample 

Teacher Rating of 
Reading Proficiency 



ELPT Reading 
Proficiency Score 




H 


A 


+ 


I 


L 


Row 

Total 


Advanced High 


(H) 


5 


2 


0 


0 


0 


7 


Advanced 


(A) 


14 


15 


4 


4 


0 


37 


Intermediate High 


(+) 


31 


72 


46 


12 


1 


162 


Intermediate 


(D 


13 


36 


76 


23 


9 


157 


Below Intermediate 


CL) 


' -2 


9 


44 


47 


13 


115 


Column Total 




65 


134 


170 


86 


23 


478 
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TABLE 3 



ELPT Listening Proficiency Scores by Teachers Ratings of 
Listening Proficiency for College Sample 



ELPT Listening 
Proficienc y Score 

Advanced High 

Advanced 

Intermediate High 

Intermediate 

Below Intermediate 



(H) 

(A) 

(+) 

(D 

(L) 



Teacher Rating of 
Listening Proficiency 



H_ 

2 
11 
22 
11 
• J 



A 

2 

11 

37 

19 

1 



+ 

1 

5 

25 

23 

3 



I 

0 

0 

5 

17 

3 



L_ 

0 

0 

1 
5 

2 



Row' 

Total 

5 

27 

90 

75 

10 



TABLE 4 

ELPT I L ;:,r in % P ™ nCienCy Sc0res by Teachers Ratings of 
Listening Proficiency for High School Sample 

Teacher Rating of 
Listening Proficiency 



Advanced 
Intermediate High 
Intermediate 
Below Intermediate 




Column Total 
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TABLE 5 

Reading Score Means, SDs, and Corrected Correlations with Course Grades 



39,4 4 8 

Nott.-College , ts, conurnmny college. Grades ate fbran ESL cine (.1/ - 2 5 SD ^ToT 
M) ^ Grad “ « far a regular freshman comtunirion enure. ,ul 



Corrected Correlation 
with Course Grade 

.53 
.05 



1 composition course fcV/= 3.3 f SD = 



TABLE 6 

Listening Score Means, SDs, and Corrected Correlations with Course Grades 




}8 41,8 

1 “ - <****■ ^ «» *> ES l 

Cohege 2 is a four-year college. Grades am fo, a regular freshen comoosiUo™ tu . 



Corrected Correlation 
with Course Grade 
49 
24 



i composition course (A/ = 3.3, SD = 
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TABLE 7 

Reading Score Means, SDs, and Corrected Correlations with GPA 



College 


n 


M 


SD 


Corrected Correlation 
with GPA 


2 


26 


36.5 


6.2 


.17 


3 


38 


24.7 


7.9 


.67 



Note. —College 2 is same four-year college as in Tables 5-6; n is larger because not all students had grades 
in English composition course. GPA M = 3. 1. SD = .73. College 3 is a community college. GPA M * 2.7, 
SD = 1.3. 
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TABLE 8 

Listening Score Mean,, SDs, and Corrected Correlations with CPA 
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Appendix 



Definition of Listening and Reading Proficiency Scales 



Definition of the Reading Proficiency Scale 



Summary of Afnha Cgdeg 

** Advanced High 

^ Advanced 

+ Intermediate High 

‘ Intermediate 

^ Below Intermediate 

Description.^ 



H Advanced High 



unfamiliar topT^^rSt^fa^'Abl^to such 45 technical reports, as well as texts that treat 

understand aspects of the target language cultur^^ere appr0priate fences as well as 

of language and of its literary style! There emerging awareness of the aesthetic pro D erties 

language. ^ ^ ^ be some misunderstanding of highly colloquial oMechnTcS 

A Advanced 



™ p.do,.™,, fMUilr pan=ra 

?f adi ” g ■** Wide ■?' «*“ id =* “ d fa« aod miaJTJc, , 

bibliographical information, social notices. cJrcnnal ulf “ SUch 45 ^ple short stories, news item^ 

texts wntten for the general reader. * ^ *”* r0Utme busmcss correspondence and simple technic^ 

+ Intermediate High 

social ncc^lZ'^^X 5 fU '' T*? "*"* th '>’ ^I wi.h basic p tr!onal md 

nan-auons, social correspondence, and simple academic texts Basic <n- a Cadin * mate . naIs indude descriptions and 
and temporal references may rely primarUy on lexical teras. Basic S r4mm4 t‘cal relations ^ may be misinterpreted 

I Intermediate 

Able to understand main ideas and some facts frnm rh. .• . 

needs. Texts have clear underlying bterai p teJcts dcaUa g «*h Personal and social 

announcements and instructions intended for a wiH* a' ' Readm S materials include messages, public 
tiungs. Some misunderstandings will occur. “dtence, and short descriptions of persons, S plac« and 

L Below Intermediate 

e- I— wood, or 

“ay be able to derive meaning from materials et< 7. wb en they are highly contextualized. At i?™** 

Wlcdse ace supportive. la * al « h “' — 
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4 ** 



Definition of the Listening Proficiency Scale 



Summary of Aloha Q^des 



Advanced High 
Advanced 
Intermediate High 
Intermediate 
Below Intermediate 



Descriptions 



H Advanced High 

t^ch^il 10 H UStaia mOS l spc , ech 111 staad " d dialect, but may not 

technical and academic reports and phSro D wJ^!f- ^ Wth '***&» or abstract topics, such 2 
awareness of culturally implied meanings. ? ^ L “ t#ner shows an emerging to fully competent 

A Advanced 

Able to understand main ideas and a . •. 

Comprehension may be uneven. Text tvn^- t a° a Va f iety t0 P ,cs beyond the immediate situation 
"'™*' tort leaures on familiar wpfa^tSs STf “ ?* r “ ■*»««« too <ra«£ 

be abk 10 ““PleKly Mow the sequence of id J L^rJS® Wh faCtUal bfer,na,io a. U*ner 
+ Intermediate High 

times and piSa'faSSta n “ ml, ' r ? f ‘«P« pertaining to different 
include interviews, short lectures, news items and « ?! a r ‘ P a!! . mam ldeas and/or details. Text tvues 
“ qUaDtity *** poorsr “ ^alicy than for the Advanced* ktem ! ^ k™'*' t0piCS ’ but com P r cbension isfcss 
I Intermediate 

social ““ 0>asic personal background and needs, 

transportation, shopping, persona! interests^'arf^r'r'w “° ple .““““toos to -toaions, lodging, 
telephone messages, simple announcements and ‘ Te ? typcs mdude fac e-to-face conversations’ 

comprehension breaks down in longer discount ° V * r ett Und «tanding is uneve^ 

L Below Iacermedlace 
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